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METHOD AND SYSTEM FOR GENERATING AN IMAGE 



TECHNICAL FIELD OF THE INVENTION 

This invention relates generally to the field of 
electro-optical devices and more specifically to a method 
and system for generating an image. 
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BACKGROUND OF THE INVENTION 

An optical sensor typically generates image 
information representing a sensed image. The image 
information may be analyzed by one or more image 
processors to generate at least a portion of the sensed 
image. A known technique for processing the image 
information may be performed by calculating and 
processing a matrix. Processing a matrix according to 
the known technique, however, consumes processing power, 
which may affect the battery life of a portable device. 
Additionally, memory may be required to store matrix 
information for the processed matrix, which may impact 
the cost and size of portable optical devices. 
Consequently, known techniques for processing image 
information may be unsatisfactory in certain situations. 



ATTORNEY ' S DOCKET 
019937 . 0400 



PATENT APPLICATION 



3 

SUMMARY OF THE INVENTION 

In accordance with the present invention, 
disadvantages and problems associated with previous 
techniques for generating an image may be reduced or 
5 eliminated. 

According to one embodiment, generating an image 
includes receiving a light at sensors, where the light is 
associated with images. A previous matrix is determined, 
where the previous matrix includes image information 

10 associated with an image. For each sensor, a current 

image data set corresponding to a current image is 
generated and a current matrix is determined using the 
previous matrix and the current image data. The current 
matrix includes image information associated with the 

15 current image. A fusion matrix is computed according to 

the current matrix of each sensor, where the fusion 
matrix initiates generation of a fused image. 

Certain embodiments of the invention may provide one 
or more technical advantages. A technical advantage of 

2 0 one embodiment may be that matrix computations may be 

reduced, resulting in lower power consumption and less 
memory usage. Another technical advantage of one 
embodiment may be that by calculating only the changes in 
the image from frame to frame, more efficient processing 

25 is performed, resulting in higher speed and, therefore, 

better resolution. Yet another technical advantage of 
one embodiment may be that by using hardware architecture 
to perform the matrix calculations and image analysis, 
overhead processing capabilities of a microprocessor may 

30 be avoided, resulting in faster processing at lower clock 

speeds. Yet another technical advantage of one 

embodiment may be that by reducing power and memory 
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consumption, an image generation device may be more 
effective as a portable unit, especially in remote areas 
where recharging batteries may be impractical. 

Certain embodiments of the invention may include 
5 none, some, or all of the above technical advantages. 

One or more other technical advantages may be readily 
apparent to one skilled in the art from the figures, 
descriptions, and claims included herein. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present 
invention and its features and advantages, reference is 
now made to the following description, taken in 
5 conjunction with the accompanying drawings, in which: 

FIGURE 1 is a block diagram of one embodiment of an 
image generation system that may be used in accordance 
with the present invention; 

FIGURE 2 is a block diagram of another embodiment of 
10 an image generation system that may be used in accordance 

with the present invention; and 

FIGURE 3 is a flowchart demonstrating one embodiment 
of a method for generating an image that may be used in 
accordance with the present invention. 
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DETAILED DESCRIPTION OF THE DRAWINGS 

Embodiments of the present invention and its 
advantages are best understood by referring to FIGURES 1 
through 3 of the drawings, like numerals being used for 
5 like and corresponding parts of the various drawings. 

FIGURE 1 is a block diagram of one embodiment of an 
image generation system 10 that may be used in accordance 
with the present invention. In general, system 10 may 
generate an image by determining a plurality of matrices 

10 that may be fused together to generate an image. An 

image processor 2 0 includes a matrix generator 24 that 
generates a matrix for each image received at each sensor 
of system 10 by detecting changes in the image from frame 
to frame, which may potentially reduce the processing 

15 workload of image processor 20. 

In operation, a sensor stage 15 includes a plurality 
of sensors that receive light 12 associated with a 
plurality of images. Image processor 20 determines a 
matrix for the light received at each sensor . A fusion 

20 processor 30 fuses the matrices to compute a fusion 

matrix that may be used by a fused image generator 4 0 to 
initiate the generation of a fused image. According to 
the illustrated embodiment, system 10 includes sensor 
stage 15, image processor 20, fusion processor 30, fused 

25 image generator 40, and a display 50 coupled as shown in 

FIGURE 1. 

Sensor stage 15 includes a plurality of sensors 
S lf ... , S n , . . . , S p/ where each sensor is operable to receive 
light 12. Sensor stage 15 may include any number of 
30 sensors S lt ...,S p that perform any suitable sensing 

without departing from the scope of the invention. For 
example, sensor stage 15 may include three sensors, S lf 
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S 2 , and S 3 , each sensor sensing light 12 at different 
wavelengths, K lf A 2 , A 3 . In that embodiment, system 10 
may be used to generate a color image based on the 
information received at sensors S 2 through S 3 if each 
5 sensor filters light 12 of wavelengths corresponding to 

the colors red, blue, and green. Sensors S lf ...,S p may 
comprise cameras, infrared receivers, image intensifier 
devices, or any other suitable light sensing device. 
Sensors S lf ...,S p may also filter any range of wavelengths 

10 without departing from the scope of the invention. 

Sensor stage 15 analyzes the sensed light 12 to 
yield raw image data. According to the illustrated 
embodiment, each sensor S n analyzes sensed light 12 
according to its filtering characteristics and outputs 

15 raw image data i n , where n represents the corresponding 

sensor. For example, a first sensor Sx analyzes light 12 
to generate raw image data i lf while a second sensor S 2 
analyzes light 12 to generate raw image data i 2 . Sensor 
S n may generate raw image data i 2 in any suitable format, 

20 such as in analog or digital format, or in any other 

suitable format without departing from the scope of the 
invention. According to one embodiment, each sensor S n 
generates raw image data i n as a stream of bits, 
packet i zed and/or encapsulated data, binary or otherwise, 

25 or in any other suitable format and at any suitable rate. 

Image processor 2 0 analyzes the raw image data i n 
received from sensor stage 15 to determine enhancements 
to the image and to prepare the image for fusion. 
According to the illustrated embodiment, image processor 

3 0 2 0 includes an image data generator 22 and a matrix 

generator 24. 
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Image data generator 22 receives the raw image data 
i n from sensor stage 15 and generates image information. 
The image information may be used by matrix generator 24 
to generate a matrix for each sensed image. According to 
5 the illustrated embodiment, image data generator 22 

generates image information I n , where n represents the 
corresponding sensor. Image information I n may include 
any data corresponding to the sensed image, such as raw 
data, functions, numbers, arrays, or any other suitable 

10 representation of image data. 

Image data I n may be received at any port, buffer, 
cache, memory, storage area, or any other suitable module 
of image data generator 22 of image processor 20. In one 
embodiment, image information I n may be buffered at a rate 

15 of a specific number of lines of data at a time, such as 

two lines of data per millisecond. Any other suitable 
buffering technique using any suitable rate may be used 
at image data generator 22 without departing from the 
scope of the invention . 

2 0 Matrix generator 24 determines a matrix M n for each 

sensed image. The matrix M n quantifies the image 

information using algorithms, operations, or functions 
that may be used to represent the image. Matrices 
Mi, ... , M p may perform the algorithms, operations, or 

2 5 functions using the corresponding image information 

I lt ...,I p to completely describe the image sensed by the 
corresponding sensor S lf . . . , S p . Matrices M lf ... ,M P may 
include, however, any other quantification or 
characterization of an image representation without 

30 departing from the scope of the invention. For example, 

matrix M n may quantify key features of an image, and may 
include target identifiers describing portions of the 
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image that are targets; functions, operations, or 
algorithms representing gradients or variances of the 
spatial frequencies of the image; histograms providing 
information about light and dark areas of the image; or 
5 any other suitable manner of describing features of an 

image that may be of interest . 

According to the illustrated embodiment, matrix 

generator 24 calculates a current matrix M x n , where n 
represents the corresponding sensor, and i represents the 
10 current image frame F±. According to one embodiment, 

current matrix M x n is calculated by determining the 
changes in the image from frame to frame. For example, 
at frame F 2 for the image sensed by sensor S lt matrix 

generator 24 determines a current matrix . At a 

15 subsequent frame F 2 , matrix generator 24 determines the 

changes in the images between frame F x and frame F 2 by 
analyzing the current image information J 2 generated by 
sensor S ± and extracting the changes between the analyzed 
current image information J 2 and the previous image 

20 characterized by a previous matrix M ( 1 1 ' 1> . 

Image processor 2 0 may compute a change in the image 
using image information I n and matrices M lt . . . ,M n . 
According to the illustrated embodiment, image processor 
2 0 may calculate a change matrix AM n indicating a change 

2 5 in the image according to the previous matrix M ( ^~ 1} and 

the current image information I n . Using the change matrix 
AiVf n , image processor 2 0 may approximate the current matrix 
by adjusting the previous matrix M ( *~ 1} according to the 
change matrix AM n . 

30 Fusion processor 30 generates a fusion matrix 

representing an enhanced image. According to the 
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illustrated embodiment, fusion matrix M f may comprise a 
matrix comprising numbers, functions, operations, 
algorithms, or any other suitable representation of the 
fusion of a plurality of matrices. According to one 
5 embodiment, fusion processor 30 receives and analyzes 

current matrices , . . . , M x n to determine image 

enhancements that may be performed to the image before 
displaying the image. Fusion processor 30 may use any 
suitable fusion technique to generate fusion matrix M f 

10 without departing from the scope of the invention. 

Fused image generator 40 may process fusion matrix M f 
and generate a display matrix 45. Display matrix 45 uses 
the fusion matrix M f and the current image information 
Ti/.../2ji to initiate the generation of a fused image at 

15 display 50. According to the illustrated embodiment, 

fused image generator 40 implements the approximations of 
the current matrices M* , . . . , M x n fused at matrix M f using 
the image information I le ..*,I n to generate a display 
matrix 45. 

2 0 According to one embodiment, fused image generator 

40 may generate display matrix 45, any fused image, or 
data representing an image for displaying at display 50 
so that a human eye can perceive an image that at least 
in part represents at least a portion of the image 
25 associated with light 12. For example, fused image 

generator 4 0 may initiate the generation of the image by 
processing display matrix 45 so that at least thirty 
frames per second are displayed at display 50 . Any other 
suitable rate of processing may be used at fused image 

3 0 generator 4 0 without departing from the scope of the 

invention. For example, fused image generator 4 0 may 
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process display matrix 45 at the rate of sixty frames per 
second . 

Display 50 may include any output device suitable 
for displaying an image. According to one embodiment, 
5 display 50 may comprise a monitor, a screen, a projection 

surface, an x-ray tube, an anode screen, or a liquid 
crystal display (LCD) , or any other device suitable for 
displaying an image. 

Modifications, additions, or omissions may be made 

10 to system 10 without departing from the scope of the 

invention. For example, image processor 2 0 may be 
implemented using separate processors for the function of 
generating a matrix and the function of generating the 
image information. As another example, fused image 

15 generator 40 and display 50 may be modified so that 

display 50 comprises a processor or other logic operable 
to perform the functions of fused image generator 40. As 
used in this document, "each" refers to each member of a 
set or each member of a subset of a set . 

2 0 Additionally, functions may be performed using any 

suitable logic comprising hardware, software, other 
logic , or any other suitable combination of the 
preceding. For example, although the term "processor" 
has been used to describe image processor 20, fusion 

2 5 processor 30, and fused image generator 40, any logic 

architecture may be used to perform the specific 
functions of each processor. According to the 

illustrated embodiment, image processor 20, fusion 
processor 30, and fused image generator 40 may comprise a 

3 0 field programmable gate array (FPGA) , a neural net, logic 

operators, hardware architecture, solid state circuits, 
other real time gate devices, or any other suitable 
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device that may be used to process the specific functions 
of image processor 20, fusion processor 30, and/or fused 
image generator 40. Any suitable number of logic devices 
may be used to perform the functions of the processors at 
5 system 10. For example, system 10 may employ hundreds or 

thousands of gates to generate an image using the 
functions performed by the processors at system 10. 
Accordingly, system 10 may generate an image using any 
hardware suitable for processing image information I 1 -I nf 

10 matrices M 1 -M n , fusion matrix M f , and display matrix 45. 

A known technique for processing the image 
information may be performed by calculating and 
processing a matrix for each frame of the sensed image. 
Processing a matrix for each frame, however, consumes 

15 processing power, which may affect the battery life of a 

portable device. Additionally, memory may be required to 
store matrix information for the entire processed matrix, 
which may impact the cost and size of portable optical 
devices. System 10 may provide an advantage over the 

20 known technique. Basing the architecture of system 10 on 

the above-mentioned hardware avoids the use of 
microprocessors, which may result in lower consumption of 
memory capacity and power. Reducing or eliminating the 
memory capacity used may also increase the processing 

25 speed at which system 10 generates an image, while 

maintaining power consumption to a minimum. 

FIGURE 2 is a block diagram of another embodiment of 
an image generation system 10 that may be used in 
accordance with the present invention. According to the 

30 illustrated embodiment, the system 10 illustrated at 

FIGURE 2 includes an enhancement processor 35 that is 
coupled to image processor 2 0 and fused image generator 
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40 as shown. Sensor 16 as illustrated is part of sensor 
stage 15 described with reference to FIGURE 1. 
Similarly, image processor 20, fusion processor 30, and 
fused image generator 4 0 are configured and operate as 
5 described with reference to FIGURE 1. 

Enhancement processor 3 5 may operate to enhance and 
condition components of matrices M lt . . . ,M n to generate 
enhanced image information T e . Enhancement processor 3 5 
may process the enhancements prior to and separately from 

10 fusing the matrices at fusion processor 30. For example, 

enhancement processor 3 5 may receive matrix M 2 and image 
information I ± to condition, enhance, or initiate the 
conditioning or enhancement of the image data represented 
by the image information J 2 and matrix M 2 . 

15 Additional enhancement processors 35 may be added to 

one or more image paths of system 10. For example, an 
additional enhancement processor 35 may be added at the 
image path from a sensor S 2 (not shown) so that image 
information I 2 and matrix M 2 may be processed in parallel 

2 0 to the processing of the image information I lf and matrix 

M x corresponding to sensor S 2 - Any suitable number of 
enhancement processors 35 may be added at system 10 
without departing from the scope of the invention. 
Additionally, although the enhancement processing of the 

25 images has been described as occurring in parallel, any 

processing timing may be used by enhancement processor 3 5 
to produce enhanced image J e . 

By enhancing an image at enhancement processor 35, 
fusion processor 30 may fuse the matrices M lf ...,M n 

30 without performing enhancement of the image content. 

Separating the functions of enhancement and fusion may 
result in a more versatile circuit capable of enhancing 
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the image prior to fusing the matrices, or otherwise, as 
long as enhanced image information I e and fusion matrix Mf 
are received in parallel at the fused image generator 40. 

Additionally, the image information I lf ...,I n and the 
5 matrices M lf . . . ,M n may be processed at a higher speed, 

which may be used if a high image resolution is desired. 
For example, enhancement processor 3 5 may be used to 
improve the quality of the fused image 55 if a display 
resolution of 1280x960 and higher is desired. 

10 Enhancement processor 35 may be used, however, regardless 

of the desired resolution. For example, enhancement 
processor 35 may be used to produce a resolution of 
640x480 at a VGA display or any other suitable display. 

Modifications, additions, or omissions may be made 

15 to system 10 without departing from the scope of the 

invention. For example, enhancement processor 3 5 may be 
included at image processor 2 0 to ensure that enhancement 
of an image is performed prior to fusing the images at 
fusion processor 30. As yet another example, a display 

2 0 5 0 may be added at system 10 of FIGURE 2 so that fused 

image 5 5 may be displayed. 

Additionally, functions may be performed using any 
suitable logic comprising software, hardware, other 
logic, or any suitable combination of the preceding. For 

25 example, as was described with reference to FIGURE 1, 

enhancement processor 35, although described as a 
processor, may be configured using the hardware logic 
described with reference to the processors of FIGURE 1. 
Enhancement processor 3 5 may, however, be configured 

30 differently from the image processor 20, fusion processor 

30, and fused image generator 40 described with reference 
to FIGURE 1. 
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FIGURE 3 is a flowchart demonstrating one embodiment 
of a method for generating an image that may be used in 
accordance with the present invention. The method begins 
at step 60 where sensors S lt ...,S n at sensor stage 15 
5 receive light 12. According to the illustrated 

embodiment, at least two sensors may be used at sensor 
stage 15 for system 10 to generate at least two 
representations of an image associated with light 12. 

The method proceeds to step 62, where matrix 

10 generator 24 establishes a previous matrix. According to 

the illustrated embodiment, a previous matrix may be 
established using the image information that may be 
received from image data generator 2 2 and/or sensor stage 
15. Any other initialization matrix may be used by 

15 matrix generator 24 to establish a previous matrix 

without departing from the scope of the invention. At 
step 64, image data generator 22 generates current image 
data associated with a current frame received at a 
sensor. For example, image data generator 22 may 

20 generate image information J 2 associated with a frame F 2 

received at sensor S ± . 

Matrix generator 24 calculates a change matrix 
according to the previous matrix and the current image 
data at step 66. As was described with reference to 

2 5 FIGURE 1, a change matrix AM n representing the changes in 

the image may be used to calculate a current matrix M x n , 
and may be calculated by extracting the differences 
between the current image data I n and the previous matrix 
M ( *~ 1} . The change matrix AM n may be calculated according 

30 to Equation (1) : 



AM n = I - M ( »~ 1} 

n n n 
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Matrix generator 24 calculates a current matrix 
according to the previous matrix and the change matrix, 
at step 68. A current matrix M x n may be described by 
5 Equation (2) : 

M< n 1} = M ( *- 1} + AM n (2) 

As was described with reference to FIGURE 1, a 
10 matrix M l n for the current frame i may be calculated by 

analyzing the image information I x to determine how to 
condition the image so that enhancements may be performed 
to the previous matrix. According to one embodiment, 
matrix generator 24 analyzes image information I n by 
15 extracting functions that may be performed on the image 

data to condition image characteristics such as spurious 
portions of an image, weak portions of an image, contrast 
aspects of an image, brightness, sharpening, dulling, or 
any other suitable image characteristic suitable for 

2 0 enhancement by image processor 20. 

At step 70 the method determines if a next sensor is 
available at system 10. If there is a next sensor, the 
method returns to step 64, where the image data generator 
22 of image processor 2 0 generates a current image data I n 
25 associated with a current frame received at the next 

sensor S 2 . 

If there is no next sensor at step 70, the method 
proceeds to step 72, where fusion processor 3 0 computes a 
fusion matrix M f according to the current matrices 

3 0 , . . . , . According to one embodiment, the fusion 
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matrix M f may be generated by fusing at least two running 
frames of data included in current matrices , . . . , M„ . 

Enhancement processor 3 5 may enhance the current 
image data associated with the received frame at step 74 . 
5 For example, as was described with reference to FIGURE 2, 

enhancement processor 35 may enhance, condition, or 
initiate the conditioning or enhancement of the image 
data to produce enhanced image information I e . 

At step 76, fused image generator 4 0 generates a 

10 fused image 55 using the fusion matrix M f . According to 

one embodiment, fused image generator 4 0 may receive 
enhanced image I e and fusion matrix M f to generate a 
display matrix 45 that may be used to generate fused 
image 55. According to another embodiment, fused image 

15 generator 4 0 may receive fused matrix Mf and image 

information I lf ...,I n to generate a display matrix 45. 
Fused image generator 40 generates display matrix 45 
using the functions included in fused matrix M f applied to 
the image information I lf . . .,I n . 

2 0 Image processor 2 0 updates the previous image matrix 

with the current image matrix at step 78 . The method 
determines if there are next frames to be processed at 
step 80. If there are frames to be processed, the method 
returns to step 74 to generate a current image data 
25 associated with a current frame received at a sensor. If 

there are no next frames to process, the method 
terminates . 

Modifications, additions, or omissions may be made 
to the method without departing from the scope of the 

3 0 invention. For example, enhancing the current image data 

associated with a received frame at step 74 may be 
omitted if an enhancement processor 35 is not used at 
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system 10. As another example, a step may be added after 
enhancing the current image data at step 74, where a 
display matrix 4 5 is generated at fused image generator 
40. Additionally, steps may be performed in any suitable 
5 order without departing from the scope of the invention. 

For example, calculating a current matrix according to 
the previous matrix and the current image data at step 66 
may be performed substantially simultaneously. As 
another example, determining if a next sensor is to be 

10 processed at step 70 may be omitted so that the steps of 

generating a current image data at step 64, calculating a 
current matrix at step 66, and calculating a change 
matrix at step 68 are performed substantially in parallel 
for each sensor S n at sensor stage 15. As yet another 

15 example, computing a fusion matrix according to the 

current matrices at step 72 may be performed 
substantially simultaneously with enhancing the current 
image data associated with a received frame at step 74 . 

Certain embodiments of the invention may provide one 

20 or more technical advantages. A technical advantage of 

one embodiment may be that matrix computations may be 
reduced, resulting in lower power consumption and less 
memory usage. Another technical advantage of one 
embodiment may be that by calculating only the changes in 

25 the image from frame to frame, more efficient processing 

is performed, resulting in higher speed and, therefore, 
better resolution. Yet another technical advantage of 
one embodiment may be that by using hardware architecture 
to perform the matrix calculations and image analysis, 

30 overhead processing capabilities of a microprocessor may 

be avoided, resulting in faster processing at lower clock 
speeds. Yet another technical advantage of one 
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embodiment may be that by reducing power and memory 
consumption, an image generation device may be more 
effective as a portable unit, especially in remote areas 
where recharging batteries may be impractical. 

Although an embodiment of the invention and its 
advantages are described in detail, a person skilled in 
the art could make various alterations, additions, and 
omissions without departing from the spirit and scope of 
the present invention as defined by the appended claims. 



